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Abstract — This  paper  addresses  real-time  weapon  classification 
by  analysis  of  asynchronous  acoustic  data,  collected  from  micro¬ 
phones  on  a  sensor  network.  The  weapon  classification  algorithm 
consists  of  two  parts:  (i)  feature  extraction  from  time-series 
data  using  Symbolic  Dynamic  Filtering  (SDF),  and  (ii)  pattern 
classification  based  on  the  extracted  features  using  Language 
Measure  (LM)  and  Support  Vector  Machine  (SVM).  The  pro¬ 
posed  algorithm  has  been  tested  on  field  data,  generated  by  firing 
of  two  types  of  rifles.  The  results  of  analysis  demonstrate  high 
accuracy  and  fast  execution  of  the  pattern  classification  algorithm 
with  low  memory  requirements.  Potential  applications  include 
simultaneous  shooter  localization  and  weapon  classification  with 
soldier-wearable  networked  sensors. 

1.  Introduction 

Sniper  attacks  have  become  one  of  the  major  sources  of 
casualty  in  asymmetric  warfare,  especially  in  urban  environ¬ 
ments.  A  counter- sniper  system  that  assists  identification  of 
the  shooter’s  location  and  weapon  class  information  would 
significantly  reduce  the  potential  peril  of  both  soldiers  and 
civilian  population.  Counter- sniper  systems  make  use  of  sev¬ 
eral  different  physical  phenomena  that  are  related  to  weapon 
data  (e.g.,  acoustic,  visual,  or  electromagnetic  signals).  In  spite 
of  the  wide  range  of  possible  measurement  devices,  acoustic 
signals  (e.g.,  muzzle  blast,  shockwave,  and  surface  vibra¬ 
tion  [1])  apparently  provide  the  most  convenient  and  accurate 
way  to  identify  sniper  shots.  Hence,  a  majority  of  existing 
counter- sniper  systems  use  acoustic  signals  as  the  primary 
information  source  [2].  In  recent  years,  several  commercial 
sniper  localization  systems  have  been  developed  [3]  [4]  [5], 
which  have  not  taken  weapon  classification  into  considera¬ 
tion.  To  this  end,  Volgyesi  et  al  [6]  developed  a  soldier- 
wearable  sensor  network  system  for  both  shooter  localization 
and  weapon  classification.  It  estimates  the  trajectory,  range, 
caliber,  and  weapon  type  using  data  from  a  single  sensor 
or  fusion  of  multiple  time- synchronized  sensors,  where  the 
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weapon  classification  is  dependent  on  the  localization.  To 
alleviate  the  requirement  of  time  synchronization,  Damarla  et 
al  [7]  developed  a  sniper  localization  method  for  a  network 
of  sensors,  which  relies  only  on  the  time  difference  of  arrival 
(TDOA)  between  the  muzzle  blast  and  shock  wave  from 
multiple  single  sensor  nodes,  relaxing  the  need  for  precise 
time  synchronization  across  the  network. 

The  goal  of  the  research  work  explored  in  this  paper  is  to 
formulate  a  real-time  weapon  classification  algorithm  based 
on  asynchronous  time  series  data  collected  from  microphones 
on  a  sensor  network.  The  major  contribution  of  this  paper 
is  formulation  of  a  weapon  classification  algorithm  that  is 
capable  of  real-time  feature  extraction  and  pattern  classifica¬ 
tion  directly  from  the  time  series  of  asynchronous  acoustic 
signals,  collected  from  networked  sensors.  This  algorithm  is 
independent  of  sniper  location  and  thus  is  expected  to  be  more 
reliable  under  adverse  conditions  in  battle  fields  than  existing 
weapon  classification  systems  [6]. 

2.  Review  of  Underlying  Mathematical  Concepts 

The  weapon  classification  algorithm  is  built  upon  the 
following  major  concepts:  (i)  Symbolic  Dynamic  Filtering 
(SDF)  [8]  [9]  for  feature  extraction;  and  (ii)  Language  Measure 
(LM)  [10][11]  and  Support  Vector  Machine  (SVM)  [12][13] 
for  pattern  classification.  While  the  theories  of  SDF,  LM,  and 
SVM  are  reported  in  details  in  existing  literature,  this  section 
briefiy  presents  the  underlying  concepts  that  have  significant 
relevance  to  this  paper. 

A.  Symbolic  Dynamic  Filtering  (SDF)  for  Feature  extraction 

Figure  1  pictorially  elucidates  the  concepts  of  partitioning 
a  finite  region  of  the  phase  space  of  the  dynamical  system 
under  consideration  and  a  mapping  from  the  partitioned  space 
into  the  symbol  alphabet,  where  the  symbols  are  indicated  by 
Greek  letters  (e.g.,  a,  /^,  7,  (5,  •  •  •  )•  h  also  shows  conversion  of 
the  symbol  sequence  into  a  probabilistic  finite- state  automaton 
(PFSA)  and  generation  of  the  state  probability  vectors  at 
the  current  and  the  reference  conditions.  The  states  of  the 
PFSA  and  the  associated  histograms  in  Fig.  1  are  indicated  by 
numerics  (i.e.,  0,  1,  2  and  3).  Feature  extraction  from  training 
data  is  comprised  of  the  following  steps. 
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Fig.  1.  Concept  of  Symbolic  Dynamic  Filtering  (SDF) 


1 )  Collection  of  Time  Series  Data:  Sensor  time  series  data, 
generated  from  a  physical  system  or  its  dynamical  model,  are 
collected  as  training  or  test  sets  over  the  range  of  operation. 
A  compact  (i.e.,  closed  and  bounded)  region  G  (i.e., 
the  n-dimensional  real  space),  where  n  G  N  (i.e.,  the  set  of 
positive  integers),  within  which  the  quasi- stationary  time  series 
is  circumscribed,  is  identified.  Let  the  space  of  time  series  data 
sets  be  represented  as  Y,  where  the  number  of  data  points 
in  each  time  series  is  sufficiently  large  for  convergence  of 
statistical  properties  within  a  specified  threshold  19].  Then, 
y*  =  {Vi^yh  •••}  ^  ^  denotes  a  time  series  for  the  data  set 
i  G  {0,1, —  1},  where  I  is  the  number  of  time  series  data 
sets  under  consideration. 

2)  Space  Partitioning:  Time  series  data  are  encoded  by 

introducing  a  partition  ®  =  {5o, ...,  that  consists 

of  m  mutually  exclusive  (i.e.,  fl  5/.  =  0  Vj  7^  k), 
and  exhaustive  (i.e.,  =  Ct)  cells.  Let  each  cell  be 

labeled  by  symbols  dj  G  S  where  S  =  {ao, ...,  cr^-i} 
is  called  the  alphabet.  This  process  of  coarse  graining  can 
be  executed  by  appropriate  partitioning  (e.g.,  uniform  or 
maximum  entropy  114])  of  the  data  set.  Then,  the  data  points 
of  the  reference  time  series  y^,  which  visit  the  cell  Bj  are 
assigned  the  corresponding  symbol  as  ay  Vj  =  0, 1, ...,  m  —  1. 
This  step  enables  transformation  of  the  reference  time  series 
y^  to  a  symbol  sequence  s^=  {5?,  52,...},  where  each  Sy 
G  H.  To  alleviate  the  difficulties  associated  with  noisy  time 
series,  symbolization  is  carried  out  by  Hilbert  transform-based 
Analytical  Signal  Space  Partitioning  (ASSP)  114],  which  is  an 
essential  ingredient  of  SDF  analysis  in  the  proposed  weapon 
classification  algorithm.  Symbol  sequences  are 

generated  from  the  respective  time  series,  y^,  •  •  •  ,  y^“^,  using 
the  same  partitioning  for  generation  of  the  symbol  sequence 

from  the  reference  time  series  y^. 

3)  Construction  of  Probabilistic  Finite  State  Automata 
(PFSA):  Probabilistic  finite  state  automata  (PFSA)  are  con¬ 
structed  18]  with  a  chosen  depth  D  and  the  corresponding 
(r  X  r)  state  transition  matrices  11*  =  [tt*^]  are  generated  by 
running  the  symbol  sequences  through  the  PFSA  structure;  the 
pair  of  subscripts  j^k  G  {1,  2, ...,  r}  denotes  a  state  transition 
from  j  to  k  and  the  superscript  i  denotes  the  training 


data  set,  i  G  {0,1, ...,/  —  1}.  Since  tt*^  >  0  is  the  transition 
probability  from  state  j  to  state  k,  H*  is  a  stochastic  matrix, 
i.e.,  =  1  Vj  G  {l,2,...,r}. 


B.  Pattern  Classification 

A  threshold-based  binary  classifier  is  constructed  in  terms 
of  a  signed  scalar  measure  110] 111]  of  the  language  generated 
by  the  PFSA  n.  The  classification  logic  is  as  follows: 


Class  = 


Cl  :  if  u  <  0 
C2  :  if  a  >  0 


(1) 


where  u  is  the  language  measure  obtained  by  assigning  a 
weight  to  each  state  of  the  PFSA  11. 

Definition  2.1:  The  characteristic  vector  %  assigns  a  signed 
real  weight  to  each  of  the  r  states  of  the  PFSA,  where  larger 
weights  are  assigned  to  relatively  more  desirable  states.  The 
(1  X  r)  characteristic  vector  is  defined  as: 


X  =  [Xl  X2  •  •  •  Xr]  (2) 

Definition  2.2:  Measure  of  the  language  generated  by  a 
PFSA  in  terms  of  its  characteristic  vector  %  is  defined  as: 

i>{0)  =  0  [/  —  (1  —  6>)n]“^x^  where  0  G  (0, 1)  (3) 

Proposition  2.1:  The  measure  i>(0)  =  lim0^o+  exists 
and  is  bounded  as:  ||i>(0)||oo  <  llxlloo- 

Proof:  Given  in  111].  ■ 

Proposition  2.2:  Given  a  primitive  (i.e.,  irreducible  and 
acyclic)  state  transition  matrix  H,  the  measure  in  Eq.  (3) 
reduces  to  i>(0)  =  z/  1  in  the  limit,  where  1  =  [1  1  ...  1]^. 
Then,  the  scalar  measure  a  is  denoted  as  [11]: 

v  =  PX^  (4) 


where  p  =  [pi,P2,  is  the  (1  x  r)  state  probability 

vector  that  is  the  (sum-normalized)  left  eigenvector  of  n 
corresponding  to  its  unique  unity  eigenvalue  115]. 

Proof:  Given  in  111].  ■ 

The  scalar  measure  a  is  of  the  form 

r  r  — 1 

i=i  i=i 

r-l  /  r-1  \ 

=  ^PiXi  +  F  “  ^^7 

r— 1  r— 1 

=  (Xi  -  Xr)  +Xr  =  +  &  (5) 

i=l  i=l 

where  =  (xi  —  Xr)  and  b  =  Xr-  Therefore,  the  scalar 
measure  u  is  an  affine  transformation  of  the  (r— 1)  independent 
state  probabilities,  where  the  rth  state  probability  may  be 
expressed  in  the  form  1  —  "f2lZi  Pi-  Further,  the  equi-measure 
surfaces  in  space  of  probability  vectors  are  hyper-planes  as 
described  by  Eq,  (5).  The  classifier  construction  involves 
computation  of  the  values  of  a^,  i  =  l,...,r  —  1  and  6, 
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Fig.  2.  Flow  chart  of  the  proposed  methodology 


that  in  turn  may  be  used  to  evaluate  the  characteristic  vector 
X  =  [xi  •••  Xr]-  The  classifier  follows  the  form  of  Eq.  (1). 

The  support  vector  machine  (SVM)  [13]  [12]  estimates  x 
and  simultaneously  maximizes  the  separation  margin  between 
the  two  classes.  In  the  linearly  separable  case,  the  binary 
SVM  classifier  selects  a  hyperplane  to  separate  the  two  classes 
by  maximizing  the  margin  that  is  defined  as  the  sum  of  the 
distances  of  the  hyperplane  to  the  closest  points  of  the  two 
classes.  If  the  two  classes  are  non-separable,  positive  slack 
variables  are  introduced  to  allow  some  of  the  training  samples 
to  fall  on  the  wrong  side  of  the  separating  hyperplane.  The 
SVM  then  finds  the  hyperplane  that  maximizes  the  margin 
and  simultaneously  minimizes  a  quantity  directly  related  to  the 
number  of  classification  errors.  This  procedure,  called  “Soft 
Margin”  method,  is  an  extension  of  the  linear  SVM  [16]. 

Figure  2  depicts  the  flow  chart  of  the  proposed  classification 
algorithm  that  is  constructed  based  on  the  theories  of  SDF, 
FM,  and  SVM.  Upon  collection  of  acoustic  time  series  data, 
symbol  sequences  are  generated  by  Analytic  Signal  Space 
Partitioning  Partitioning  (ASSP)  [14]  that  is  invariant  for  both 
training  and  testing  stages.  In  the  training  stage,  I  train  sets  of 
rifie-shot  time  series  data  with  known  class  labels  are  selected. 
A  PFSA  structure  is  constructed  using  SDF;  subsequently,  a 
feature  vector/;*,  i  G  {0, ...,  krain  —  ^}  is  generated  for  each  set 
of  the  rifie-shot  data.  The  time  series  data  belongs  to  exactly 
one  of  the  two  known  classes  Ci  and  C2. 

The  feature  vectors  are  inputs  to  the  FM  &  SVM  module 
that  generates  a  hyperplane  that  maximizes  the  margin  and 
minimizes  classification  errors  between  feature  vectors  of  the 
training  data.  A  linear  kernel  has  been  used  in  this  paper.  In 
the  testing  stage,  the  feature  vectors  p\  i  G  {0, ...,  kest  —  1} 
are  generated  from  SDF  with  unknown  class  labels,  and  are 
then  separated  by  the  hyperplane  obtained  in  the  training  stage. 
The  SM  &  SVM  algorithm  yields  a  binary  output  (i.e.,  Ci  or 
C2)  as  the  class  labels  of  the  testing  data. 

3.  Results  OF  Field  Data  Analysis 

Figure  3  shows  a  data  collection  scenario  for  classification 
of  two  weapon  types,  namely,  Rifie  1  and  Rifie  2  that  were 
fired  from  two  different  locations  that  are  ^  60m  apart.  Eight 
microphone  sensors  are  distributed  over  a  region  of  ^  30m  x 
30m  around  each  of  the  three  aim-points  that  are  ^  250m 
down  range. 

For  each  of  the  three  aim-points,  the  data  set  of  Rifie  1, 
which  is  generated  by  firing  Rifie  1  from  location  B,  has  been 


Fig.  3.  A  data  collection  scenario  with  8  microphones  and  2  rifle  locations 
used  for  partitioning;  these  data  sets  are  not  used  in  the  training 
stage  or  testing  stage.  Referring  to  Section  2,  a  maximum- 
entropy  analytic  signal  space  partitioning  (ASSP)  [14]  is 
generated  in  the  radial  direction  with  {'ErI  =  4  segments 
and,  in  the  angular  direction,  with  |F^|  =  1  segment  [14]; 
thus,  the  alphabet  size  is  |I1|  =  IF^I  x  \Er\  =  4.  Thus, 
for  each  aim-point,  there  is  a  unique  partition  that  is  kept 
invariant  in  both  training  and  testing  stages.  For  construction 
of  PFSA  from  symbol  sequences,  a  depth  of  D  =  1  is 
found  to  be  adequate  to  capture  sufficient  information  for 
pattern  classification  in  this  application  [8].  Each  feature  vector 
obtained  as  the  (stationary)  state  probability  vector  of  the 
PFSA  represents  the  respective  signature  of  the  rifie  acoustic 
signal.  The  top  row  of  Fig.  4  exhibits  a  typical  time  series 
of  acoustic  data  from  Rifie  1  and  Rifie  2,  which  are  collected 
from  the  same  microphone;  examples  of  feature  vectors  for  the 
two  types  of  rifies  are  shown  as  histograms  in  the  bottom  row 
of  Fig.  4,  which  display  how  the  structure  of  the  underlying 
probability  distribution  varies  with  the  rifie  type. 
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Fig.  4.  Acoustic  signals  and  respective  feature  vectors 


Three  plots,  arranged  as  a  vertical  column  in  Fig.  5,  present 
the  results  of  classification  for  two  types  of  rifies  firing  from 
location  A  at  aim-points  1,  2  and  3,  respectively,  based  on  the 
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time  series  data  of  Microphone  4.  No  occurrence  of  pattern 
classification  errors  is  observed  in  these  experiments  for  this 
specific  choice  of  feature  extraction  and  pattern  classification 
parameters.  For  each  aim-point,  the  respective  plot  in  Fig.  5 
exhibits  the  results  of  classification  for  a  single  set  of  Rifie 
2  data  and  a  combination  of  two  different  sets  of  Rifie  1 
data  that  have  been  collected  on  two  different  days.  For  each 
aim-point,  about  a  half  of  the  data  in  each  set  are  used  for 
training  the  SVM  classifier  and  the  remaining  data  for  testing 
the  classification  algorithm.  As  stated  earlier,  the  partition  of 
individual  data  sets  for  each  aim-point  is  generated  based  on 
the  respective  data  set  of  Rifie  1  fired  from  the  building  as 
indicated  by  firing  location  B  in  Fig.  3.  In  Fig.  5,  the  feature 
extracted  from  each  data  set  is  represented  by  a  vector  that 
belongs  to  the  4-dimensional  real  space  because  |i;|  =  4 
and  D  =  1.  Since  the  elements  of  each  feature  vector  are 
stationary  probabilities  of  the  four  states  of  a  PFSA,  the 
sum  of  the  (positive)  elements  of  each  feature  vector  is  unity 
(i.e.,  belonging  to  the  3-dimensional  simplex).  Therefore,  only 
three  elements  of  the  feature  vector  are  linearly  independent, 
implying  that  the  decision  space  is  3 -dimensional  in  this 
setting.  For  each  of  the  three  plots  in  Fig.  5,  the  2-dimensional 
hyperplane  unambiguously  separates  the  patterns  of  Rifie  2 
from  those  of  Rifie  1 .  In  this  way,  a  time  series  of  rifie  data 
is  reliably  tested  for  identification  of  the  unknown  rifie  type, 
namely,  Rifie  1  or  Rifie  2  by  observing  to  which  side  of  the 
hyperplane  the  feature  vector  belongs. 

The  execution  time  of  the  algorithm  for  each  plot  in 
Fig.  5  is  less  than  1.5  seconds  on  a  desktop  computer,  which 
demonstrates  its  real-time  execution  capability. 

A.  Channel  Ejfects  on  Classification  Results 

This  section  presents  a  summary  of  channel  effects  (i.e. 
the  degradation  of  acoustic  information  due  to  propagation 
through  the  atmosphere)  on  the  classification  results,  which  are 
attributed  by:  (i)  variations  in  the  recorded  acoustic  signals  due 
to  relative  positions  of  microphones  and  calibration  settings; 
and  (ii)  Environmental  conditions  such  as  vegetation,  terrain, 
and  urban  buildings  that  infiuence  the  channel.  While  the  first 
cause  is  mainly  a  hardware  issue  that  can  be  mitigated  by 
appropriate  calibration,  the  second  cause  (i.e.,  environmental 
effects)  is  the  focus  of  this  section. 

In  real-world  applications  (e.g.,  an  urban  environment  with 
tall  buildings  and  various  other  objects),  the  shock  wave  and 
the  muzzle  blast  from  sniper  firing  are  subject  to  refiection, 
attenuation,  absorption,  diffraction,  and  other  wave  modifica¬ 
tions  as  they  propagate  through  the  atmosphere.  A  microphone 
placed  in  the  vicinity  of  an  aim-point  receives  pressure  waves 
arriving  directly  from  the  source  and  waves  arriving  later 
from  other  directions  due  to  refiections  and  scattering.  At 
distances  far  from  the  rifie  shot  trajectory,  the  shock  wave 
is  expected  to  disperse  sufficiently  by  spatial  spreading  such 
that  it  may  no  longer  be  detectable  compared  to  the  ambient 
noise  [1].  The  situation  becomes  much  more  complicated  if  the 


(a)  Aim-point  1 


(b)  Aim-point  2 


(c)  Aim-point  3 

Fig.  5.  Rifle  Classification  at  aim-points  1,  2,  and  3  (Microphone  #  4). 

surroundings  include  obstacles  and  refiecting  surfaces  so  that 
the  received  acoustic  signal  contains  multipath  interference, 
diffraction  effects,  and  other  propagation-related  fiaws.  In 
essence,  the  environmental  effects  could  be  totally  different 
at  different  sensor  locations.  An  ideal  weapon  classification 
system  should  be  independent  of  the  channel  effects  due  to 
environmental  variations. 

The  results  presented  earlier  in  this  section  make  use 
of  the  acoustic  time  series  data  from  a  single  microphone, 
namely.  Microphone  4  for  weapon  classification.  This  sub¬ 
section  reports  the  results  obtained  based  on  the  data  from 
all  8  microphones,  including  Microphone  4,  to  investigate  the 
impact  of  environmental  effects  on  the  weapon  classification. 
The  microphones  are  placed  in  different  locations  and  have 
varying  levels  of  echo/refiection  due  to  their  slightly  different 
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TABLE  I 

Rifle  Classification  Results  of  All  Microphones 


Microphone  # 

Aim-point  1 

Aim-point  2 

Aim-point  3 

Microphone  1 

95.83% 

100.00% 

83.33% 

Microphone  2 

100.00% 

100.00% 

100.00% 

Microphone  3 

100.00% 

100.00% 

95.83% 

Microphone  4 

100.00% 

100.00% 

100.00% 

Microphone  5 

91.67% 

100.00% 

100.00% 

Microphone  6 

91.67% 

100.00% 

100.00% 

Microphone  7 

91.67% 

100.00% 

100.00% 

Microphone  8 

91.67% 

91.67% 

100.00% 

Average 

97.22% 

environment.  Table  I  summarizes  the  classification  results 
obtained  from  all  8  microphones  for  each  of  the  3  aim-points. 
The  total  number  of  tests  (i.e.,  rifie  shots)  for  each  aim- 
point  is  24,  and  the  classification  success  rate  is  calculated 
by  subtracting  the  ratio  of  the  number  of  false  classifications 
over  the  total  test  number  from  1.  It  is  seen  in  Table  I  that 
a  majority  of  the  microphones  have  a  very  high  classification 
success  rate.  The  classification  success  in  Microphone  1  is 
slightly  lower  than  that  in  other  microphones  due  to  improper 
calibration  as  recorded  in  the  original  log  of  the  field  test.  The 
average  rate  of  successful  classification  is  97.22%,  which  is 
a  clear  indication  of  its  reliability  and  effectiveness.  Trade-off 
between  probabilities  of  successful  and  false  classification  is 
a  topic  of  future  research  and  is  not  addressed  in  this  paper. 

The  quality  of  classification  could  be  further  improved  by: 
(i)  increasing  the  number  of  states  in  the  PFSA,  and  (ii)  con¬ 
verting  the  linear  hyperplane  in  the  SVM  into  a  hypersurface 
by  an  appropriate  nonlinear  transformation. 

4.  Summary,  Conclusions  and  Future  Work 

This  paper  presents  a  data-driven  method  of  real-time 
weapon  classification  based  on  time  series  data  collected  from 
asynchronous  microphones  on  a  sensor  network.  The  pattern 
classification  algorithm  is  feature-based  in  the  sense  that  it 
first  converts  the  raw  time  series  data  into  a  feature  vector  of 
significantly  lower  dimension,  and  then  pattern  classification  is 
performed  based  on  the  extracted  feature  vectors.  The  feature- 
based  approach  is  well-suited  to  the  weapon  classification 
problem  because  direct  usage  of  a  large  volume  of  raw  data 
with  unknown  or  partially  known  noise  statistics  is  inefficient 
from  both  analytical  and  computational  perspectives.  Based  on 
the  data  set  analyzed  in  this  study,  the  proposed  classification 
algorithm  appears  to  be  nearly  channel  independent,  which 
renders  it  potentially  reliable  and  effective  in  the  real-world 
applications. 

The  discipline  of  data-driven  weapon  classification  is  rela¬ 
tively  new  and  requires  further  theoretical  and  experimental  re¬ 
search.  In  this  context,  the  following  topics  are  recommended 


for  future  research  before  execution  of  a  field  application  of 

the  proposed  weapon  classification  algorithm: 

•  Extension  of  the  current  algorithm  to  multi-class  pattern 
classification  with  advanced  SVM  tools. 

•  Enhancement  of  classification  performance  through  usage 
of  multi- sensor  information  fusion. 

•  Investigation  of  the  effects  of  signal-to-noise  ratio 
and  clutter  parameters  for  automatic  target  recognition 
(ATR)  [17].  In  this  context,  while  making  a  trade-off 
between  probability  of  false  alarms  and  probability  of 
successful  detection,  additional  costs  related  to  weapon 
localization  could  be  included  in  the  composite  cost 
functional,  which  will  augment  the  standard  Receiver 
Operating  Characteristics  (ROC)  curve  to  a  higher  di¬ 
mensional  Pareto  surface. 
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